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TITLE OF THE INVENTION 



Gesture Activated Home Appliance 



FIELD OF THE INVENTION 



The present invention relates to gesture recognition systems. More 
particularly, the present invention relates to systems and methods for operating a home 
appliance using gesture recognition. 



world. Today, many devices incorporate voice activation technology so certain functions of 
the device can be performed based on voice commands. For example, many home appliances 
are equipped with voice-activated technology as to allow a consumer to orally command, for 
example, a lighting system to power on by using voice commands. Such a feature is 
particularly advantageous when a person cannot manually activate a device because their 
hands are occupied or the device switch is in an inconvenient place. 



manually activate the light switch, consequently, if the lighting system in the house has voice 
activated technology therein, the person may simply say, for example, "lights on" to activate 
the lights. This technology saves the person the trouble from having to put the groceries 
down, manually activate the lights, and then pick up the groceries. The ability to use voice- 
activated technology, of course, presumes that the user is able to speak. Consequently, those 
persons who cannot speak cannot take advantage of voice-activated technology. 



Gesture recognition is not based on voice commands but, rather, allows a device to recognize 
certain gestures. The inventors, however, are not aware of any gesture recognition systems 
that are used in conjunction with the operation of home appliances. Therefore, there is a need 
for systems and methods that use gesture recognition to operate home appliances. 



BACKGROUND OF THE INVENTION 



Voice activated technology is a rapidly developing area of the computer 



For example, one may be carrying groceries into a house and is unable to 



Gesture recognition is another rapidly developing area in the computer world. 
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BRIEF SUMMARY OF THE INVENTION 
The present invention satisfies the aforementioned need by providing systems 
and methods for using gesture recognition to operate a home appliance. In one embodiment 
of the present invention, an apparatus recognizes a gesture and then causes a home appliance 
5 operation associated with the gesture to be performed. For example, a first gesture might be 
associated with powering on a home appliance and a second gesture might be associated with 
powering off the home appliance. In this manner, a user can simply gesticulate to operate a 
home appliance rather than operating the appliance manually. 

Apparatus according to the invention for operating a home appliance can 
10 include an image receiver, an image processor, an operations processor, and an appliance 
^ controller. The image receiver, which can be a video camera, for example, receives a 

continuous stream of images of a gesture performed within the image receiver's field of view 
Lfjl! and provides a video signal to the image processor. The video signal contains information 

corresponding to the continuous stream of gesture images. The image processor receives the 
15 video signal and recognizes the gesture based on the information contained in the video 
signal. 

The image processor can include an image sampler and an image comparator. 
The image sampler can be used for sampling the continuous stream of gesture images to form 
a discrete sequence of gesture images. The image comparator can then compare the discrete 
20 sequence of gesture images to each of one or more stored discrete gesture sequences, and 
recognize the gesture based on the comparison. 

The operations processor is coupled to the image processor for identifying a 
home appliance operation associated with the gesture. The operations processor can identify 
the home appliance operation by comparing the recognized gesture with each of a predefined 
25 set of gestures. Each of the predefined gestures is associated with a respective home 
appliance operation. 

The appliance controller is coupled to the operations processor for causing the 
home appliance operation to be performed. The apparatus can include a switch that is 
coupled to the appliance controller and is adapted to be coupled to the home appliance. The 
30 switch is responsive to the appliance controller for causing the home appliance operation to 
be performed. The appliance controller can be a voltage generator for generating a voltage 
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signal, or a pulse generator for generating a voltage pulse, that causes the home appliance 
operation to be performed. 

The apparatus can alsjMficlude a performance indicator for providing an 
indication that the home agpkSnce operation has been performed, a data store, readable by 
the image process^Ffror storing data representative of the gesture and of the home appliance 
operation. 



BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 

Other features of the present invention are further apparent from the following 
10 detailed description of the embodiments of the present invention taken in conjunction with 
the accompanying drawings, of which: 

FIG. 1 is a block diagram of a device used to operate a home appliance in 
accordance with the present invention; 

FIGS. 2A and 2B illustrate the device of FIG. 1 connected to home appliances; 

15 and 

FIG. 3 is flowchart illustrating a method of operating a home appliance in 
accordance with the present invention. 

DETAILED DESCRIPTION OF THE PRESENT INVENTION 

20 The present invention is directed to systems and methods for operating a home 

appliance using the principles of gesture recognition technology. In this manner, a person 

may turn on an oven, for example, by performing a recognizable gesture associated with 

turning on the oven. Similarly, the person may also turn off the oven, set the oven 

temperature, turn on the oven light, or cause the oven to perform any other operation by 

25 performing a gesture associated with respective operation. Consequently, a user can 

gesticulate to operate a home appliance rather than manually activate a switch or a button to 

operate the home appliance. 

The present invention relates to apparatuses and methods for operating a home 

appliance. The apparatus may be a computer, in whole or in part, specially constructed to 

30 perform the method of the invention, or it may be a general purpose computer selectively 

activated or reconfigured by a computer program stored in the computer. General purpose 

computing machines may be used with programs written in accordance with the teachings 
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herein, or a specialized computing apparatus may be constructed to perform the method 
described herein. 

FIG. 1 is a block diagram of a device 100 for operating a home appliance (not 
shown) using gesture recognition in accordance with the present invention. As shown, the 
5 device 100 includes an image receiver 1 10 for receiving a continuous stream of gesture 
images. The image receiver 1 10 may be any PC-based camera or the like without departing 
from the principles of the present invention. The image receiver 1 10 transmits the video 
signals to an image processor 120 by way of a cable 1 12. The video signals may be sent in 
any format such as, for example, MPEG3 format or the like without departing from the 
10 principles of the present invention. 

The image processor 120 forms a discrete sequence of gesture images based 
on the continuous stream of gesture images it receives from the image recdver 110. In this 

Uir 

%Q| manner and in one embodiment of the present invention, the image processor contains an 

t% 

image sampler 120A for forming a discrete sequence of gesture images from the received 
15 gesture images by sampling the continuous stream of images. In other words, the image 
sampler 120 A segments the continuous stream of images into a series of frames or 
"snapshots." 

The image processor 120 also derives positional data of the gestures in each 

frame, and compiles the positional data into a respective frame data set. The positional data 

20 in the frame data set is compared, in the image comparator 120B, to each of a set of stored 

sequences of positional data making up one or more recognizable gestures i.e, gestures 

already known to the system. Any gesture, such as, for example, a gesture associated with 

American Sign Language can be stored in a data store 150. In this regard, the comparator 

120B can search the data store 150 to match the received gesture with a stored gesture by 

25 using the positional data of the data frame set of the received gesture to the positional data of 

one of the known sequences stored in the data store 150. The image comparator 120B, based 

on the comparison, determines whether the user within the field of view 1 15 of the image 

receiver 1 10 has performed a recognizable gesture. 

Thus, the image receiver 1 10 can receive a continuous image of a gesture, 

30 which can be, for example, a person moving an arm horizontally across the chest. The image 

sampler 120A segments the gesture into frames. The image processor 120 then obtains 

positional data from each frame of the gesture sequence and compiles the data in a data frame 
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set. Then, the image comparator 120B compares the positional data of the data frame set to 
the positional data of the recognizable gestures stored in the database. 

Once a gesture is recognized (i.e., the system determines that the received 
image corresponds to a recognizable gesture), an operations processor 130 connected to the 
5 image processor 120 identifies a predefined home appliance operation associated with the 
recognized gesture. In one embodiment of the present invention, the operations processor 
130 determines, based on the recognized gesture, which of a plurality of predefined home 
appliance operations is to be performed. In this manner, the operations processor can access 
the data store 150 to determine the predefined home appliance operation associated with the 
10 recognized gesture. 
^2 As shown and for example, one recognizable gesture 1 16 might include 

fp motioning the letter "z" with the index finger of the hand. In this regard, once the device 100 

ih 

^ recognizes this gesture, the operations processor 130 searches the data store 150 to determine 

yi which home appliance operation is associated with motioning the letter "z" with the index 

15 finger. For example, this gesture might be associated with turning on the appliance. Another 



%f recognizable gesture, such as moving one arm horizontally across the chest, for example, 

fyi' might be associated with turning off the home appliance. 

|{f An appliance controller 140 is connected to the operations processor 130 for 

J»£t causing the predefined home appliance operation to be performed. In one embodiment of the 

20 present invention, the appliance controller is connected to a switch 180. In this manner, for 

example, if the predefined home appliance operation is turning on the appliance, the 

appliance controller 140 can be configured to transmit an electrical signal that actuates switch 

180 that turns on the appliance. Consequently, the appliance controller 140 is configured to 

actuate the power switch 180 that turns on the appliance if the gesture associated with 

25 powering on the appliance is performed and recognized by the device 100. In this regard, the 

appliance controller 140 may be a voltage generator for generating a voltage signal that 

actuates the power switch 180. Alternatively, the appliance controller 140 can be a pulse 

generator for generating a voltage pulse to actuate a switch 180 that may or may not be 

located within the appliance. 

30 The device 100 in accordance with the present invention can also include a 

performance indicator 160, such as, for example, a sound generator connected to the 

appliance controller 140. The performance indicator 160 indicates the predefined home 
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appliance operation has been performed. In one embodiment of the present invention, the 
performance indicator 160 is a sound generator that emits, in response to the appliance 
controller 140 causing the predefined home appliance operation to be performed, a sound to 
notify the user that the home appliance operation has occurred. This method of indication is 
5 particular useful if the home appliance operation cannot be visually verified, e.g. turning on 
an oven or the like. 

In one embodiment of the invention, each home appliance operation is 
associated with a different audible sound. For example, immediately after turning on the 
appliance, the sound generator may emit a single beep. Conversely, immediately after 
*J 10 turning off the appliance, the sound generator may emit a double beep. This notification 
# method provides the user with an audible indication of what particular home appliance 

I pi: operation has been performed. 

^ FIGS. 2A and 2B each illustrates the apparatus of FIG. 1 connected to a home 

fjfc appliance. As shown in FIG. 2A, lighting system 220 is connected to an apparatus 100 in 

^ 15 accordance with the present invention. As described above, the apparatus 100 receives 

images by way of image receiver 1 10 and transmits the images to an image processor 120 

through cable 1 12. The image processor 120 recognizes a sequence of gesture images based 

on the stream of gesture images received by the image receiver 1 10. 

Once a gesture is recognized, an operations processor 130 connected to the 

20 image processor 120 identifies a predefined home appliance operation associated with that 

recognized gesture. An appliance controller 140 is connected to the operations processor 130 

for causing the predefined home appliance operation to be performed. In this manner, for 

example, if the home appliance operation is turning on the lighting system 220, the appliance 

controller 140 is configured to turn on the lighting system 320 using switch 229. 

25 Consequently, the appliance controller 140 transmits a signal by way of cable 230 to the 

lighting system 220 if the gesture associated with powering on the lighting system 220 is 

performed and recognized. A performance indicator 160 connected to the appliance 

controller 140 emits, in response to the appliance controller 140 turning on the lighting 

system 220, an indication to notify the user that the lighting system 220 is on. 

30 Similarly, as shown in FIG. 2B, an oven 260 is connected to an apparatus 100 

in accordance with the present invention by cable 265. Again, as described above, the 

apparatus 100 receives images of a gesture 1 16 by way of image receiver 1 10 and transmits 
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the images to image processor 120 through cable 1 12. Image processor 120 recognizes a 
sequence of gesture images based on the stream of gesture images received by the image 
receiver 110. 

Once a gesture is recognized, an operations processor 130 connected to the 
5 image processor 120 identifies which predefined home appliance operation is associated with 
the recognized gesture. An appliance controller 140 is connected to the operations processor 
130 for causing the predefined home appliance operation associated with the recognized 
gesture to be performed. In this manner, for example, if the home appliance operation is 
turning off the oven 260, the appliance controller 140 is configured to turn off the oven 260 

10 by way of switch 264. Consequently, the appliance controller 140 causes the oven 260 to be 
turned on if the gesture associated with powering on the oven 260 is performed and 
recognized. A performance indicator 160 connected to the appliance controller 140 indicates, 
after the appliance controller 140 is turned on, that the oven 260 is on. 

The operations processor 130 can also include programming and logic that 

15 enable it to provide more sophisticated signals to the home appliance. For example, where 
the home appliance is an oven, the operations processor can be programmed to send any of a 
number of various signals to the oven, thereby causing the oven to perform any number of 
operations. For example, a first recognizable gesture can be associated with turning the oven 
on, a second gesture can be associated with setting the oven temperature, a third gesture can 

20 be associated with setting a cooking timer, a fourth gesture can be associated with turning on 
the oven's internal light, and a fifth gesture can be associated with turning the oven off. Each 
such recognizable gesture would be associated with the corresponding operation in the 
system's data store. 

Depending on the gesture that i^fecognized, the operations processor can send 

a signal to the oven's computer that enmldtes the signal the oven's computer would have 

received is the user had pushed the/dppropriate buttons on the oven itself. For example, 

pushing the oven on button mi^it cause a certain signal to be sent to the oven's computer. 

The oven's computerwcJuld then turn the oven on. Accordingly, the operations processor of 

the invention cajj/^end the same signal directly to the oven's computer, thereby causing the 

30 oven's compmer to turn the oven on. 

FIG. 3 is flowchart illustrating a method of operating a home appliance in 

accordance with the present invention. In the method, a user performs a gesture within the 
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field of view of an image receiver, such as a camera. Thus, the image receiver receives a 
continuous stream of gesture images at step 300. The system uses the continuous image 
stream to determine whether the gesture that the user is making is one of a set of one or more 
recognizable gestures. In one embodiment of the present invention, the gesture is recognized 
by first forming a discrete sequence of gesture images from the continuous image stream at 
step 310. Then, using positional data in a data frame set derived from each frame, the 
discrete sequence of gesture images is compared, at step 320, to a set of one or more stored 
sequences of discrete gesture images. Based on the comparison, the system determines, at 
step 330, whether the performed gesture is one of the set of recognizable gestures. 



appliance operation. The recognized gesture is compared to a set of stored gestures, each 
gesture being associated with a predefined home appliance operation. Once the predefined 
operation identified with the recognized gesture is found, the system causes the predefined 
home appliance operation to be performed at step 350. At step 360, an indication can be 
provided to indicate to the user that the operation has been performed. 



appliance using gesture recognition technology. Though the invention has been described 
with reference to a device that is external and connectable to a home appliance, it is 
contemplated that such a device may be integrated into such an appliance. Consequently, the 
device in accordance with the principles of the present invention should not be limited to a 
device that is separate from the home appliance. Those skilled in the art will appreciate that 
numerous other changes and modifications can be made to the preferred embodiments of the 
invention, and that such changes and modifications can be made without departing from the 
spirit of the invention. It is intended, therefore, that the appended claims cover all such 
equivalent variations as fall within the true spirit and scope of the invention. 



At step 340, the recognized gesture is identified with a predefined home 



Thus, there have been described systems and methods for operating a home 
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